Fusing modalities by multiplexed graph neural networks for outcome prediction from medical data and beyond.

With the emergence of multimodal electronic health records, the evidence for diseases, events, or findings may be present across multiple modalities ranging from clinical to imaging and genomic data. Developing effective patient-tailored therapeutic guidance and outcome prediction will require fusing evidence across these modalities. Developing general-purpose frameworks capable of modeling fine-grained and multi-faceted complex interactions, both within and across modalities is an important open problem in multimodal fusion. Generalized multimodal fusion is extremely challenging as evidence for outcomes may not be uniform across all modalities, not all modality features may be relevant, or not all modalities may be present for all patients, due to which simple methods of early, late, or intermediate fusion may be inadequate. In this paper, we present a novel approach that uses the machinery of multiplexed graphs for fusion. This allows for modalities to be represented through their targeted encodings. We model their relationship between explicitly via multiplexed graphs derived from salient features in a combined latent space. We then derive a new graph neural network for multiplex graphs for task-informed reasoning. We compare our framework against several state-of-the-art approaches for multi-graph reasoning and multimodal fusion. As a sanity check on the neural network design, we evaluate the multiplexed GNN on two popular benchmark datasets, namely the AIFB and the MUTAG dataset against several state-of-the-art multi-relational GNNs for reasoning. Second, we evaluate our multiplexed framework against several state-of-the-art multimodal fusion frameworks on two large clinical datasets for two separate applications. The first is the NIH-TB portals dataset for treatment outcome prediction in Tuberculosis, and the second is the ABIDE dataset for Autism Spectrum Disorder classification. Through rigorous experimental evaluation, we demonstrate that the multiplexed GNN provides robust performance improvements in all of these diverse applications.

Hierarchical graph representations in digital pathology.

Cancer diagnosis, prognosis, and therapy response predictions from tissue specimens highly depend on the phenotype and topological distribution of constituting histological entities. Thus, adequate tissue representations for encoding histological entities is imperative for computer aided cancer patient care. To this end, several approaches have leveraged cell-graphs, capturing the cell-microenvironment, to depict the tissue. These allow for utilizing graph theory and machine learning to map the tissue representation to tissue functionality, and quantify their relationship. Though cellular information is crucial, it is incomplete alone to comprehensively characterize complex tissue structure. We herein treat the tissue as a hierarchical composition of multiple types of histological entities from fine to coarse level, capturing multivariate tissue information at multiple levels. We propose a novel multi-level hierarchical entity-graph representation of tissue specimens to model the hierarchical compositions that encode histological entities as well as their intra- and inter-entity level interactions. Subsequently, a hierarchical graph neural network is proposed to operate on the hierarchical entity-graph and map the tissue structure to tissue functionality. Specifically, for input histology images, we utilize well-defined cells and tissue regions to build HierArchical Cell-to-Tissue (HACT) graph representations, and devise HACT-Net, a message passing graph neural network, to classify the HACT representations. As part of this work, we introduce the BReAst Carcinoma Subtyping (BRACS) dataset, a large cohort of Haematoxylin & Eosin stained breast tumor regions-of-interest, to evaluate and benchmark our proposed methodology against pathologists and state-of-the-art computer-aided diagnostic approaches. Through comparative assessment and ablation studies, our proposed method is demonstrated to yield superior classification results compared to alternative methods as well as individual pathologists. The code, data, and models can be accessed at https://github.com/histocartography/hact-net.

Learning-based defect recognition for quasi-periodic HRSTEM images.

Controlling crystalline material defects is crucial, as they affect properties of the material that may be detrimental or beneficial for the final performance of a device. Defect analysis on the sub-nanometer scale is enabled by high-resolution scanning transmission electron microscopy (HRSTEM), where the identification of defects is currently carried out based on human expertise. However, the process is tedious, highly time consuming and, in some cases, yields ambiguous results. Here we propose a semi-supervised machine learning method that assists in the detection of lattice defects from atomic resolution HRSTEM images. It involves a convolutional neural network that classifies image patches as defective or non-defective, a graph-based heuristic that chooses one non-defective patch as a model, and finally an automatically generated convolutional filter bank, which highlights symmetry breaking such as stacking faults, twin defects and grain boundaries. Additionally, we suggest a variance filter to segment amorphous regions and beam defects. The algorithm is tested on III-V/Si crystalline materials and successfully evaluated against different metrics and a baseline approach, showing promising results even for extremely small training data sets and for noise compromised images. By combining the data-driven classification generality, robustness and speed of deep learning with the effectiveness of image filters in segmenting faulty symmetry arrangements, we provide a valuable open-source tool to the microscopist community that can streamline future HRSTEM analyses of crystalline materials.

Reducing annotation effort in digital pathology: A Co-Representation learning framework for classification tasks.

Classification of digital pathology images is imperative in cancer diagnosis and prognosis. Recent advancements in deep learning and computer vision have greatly benefited the pathology workflow by developing automated solutions for classification tasks. However, the cost and time for acquiring high quality task-specific large annotated training data are subject to intra- and inter-observer variability, thus challenging the adoption of such tools. To address these challenges, we propose a classification framework via co-representation learning to maximize the learning capability of deep neural networks while using a reduced amount of training data. The framework captures the class-label information and the local spatial distribution information by jointly optimizing a categorical cross-entropy objective and a deep metric learning objective respectively. A deep metric learning objective is incorporated to enhance the classification, especially in the low training data regime. Further, a neighborhood-aware multiple similarity sampling strategy, and a soft-multi-pair objective that optimizes interactions between multiple informative sample pairs, is proposed to accelerate deep metric learning. We evaluate the proposed framework on five benchmark datasets from three digital pathology tasks, i.e., nuclei classification, mitosis detection, and tissue type classification. For all the datasets, our framework achieves state-of-the-art performance when using approximately only 50% of the training data. On using complete training data, the proposed framework outperforms the state-of-the-art on all the five datasets.

Cloud-Based Evaluation of Anatomical Structure Segmentation and Landmark Detection Algorithms: VISCERAL Anatomy Benchmarks.

Variations in the shape and appearance of anatomical structures in medical images are often relevant radiological signs of disease. Automatic tools can help automate parts of this manual process. A cloud-based evaluation framework is presented in this paper including results of benchmarking current state-of-the-art medical imaging algorithms for anatomical structure segmentation and landmark detection: the VISCERAL Anatomy benchmarks. The algorithms are implemented in virtual machines in the cloud where participants can only access the training data and can be run privately by the benchmark administrators to objectively compare their performance in an unseen common test set. Overall, 120 computed tomography and magnetic resonance patient volumes were manually annotated to create a standard Gold Corpus containing a total of 1295 structures and 1760 landmarks. Ten participants contributed with automatic algorithms for the organ segmentation task, and three for the landmark localization task. Different algorithms obtained the best scores in the four available imaging modalities and for subsets of anatomical structures. The annotation framework, resulting data set, evaluation setup, results and performance analysis from the three VISCERAL Anatomy benchmarks are presented in this article. Both the VISCERAL data set and Silver Corpus generated with the fusion of the participant algorithms on a larger set of non-manually-annotated medical images are available to the research community.

Three-dimensional solid texture analysis in biomedical imaging: review and opportunities.

Three-dimensional computerized characterization of biomedical solid textures is key to large-scale and high-throughput screening of imaging data. Such data increasingly become available in the clinical and research environments with an ever increasing spatial resolution. In this text we exhaustively analyze the state-of-the-art in 3-D biomedical texture analysis to identify the specific needs of the application domains and extract promising trends in image processing algorithms. The geometrical properties of biomedical textures are studied both in their natural space and on digitized lattices. It is found that most of the tissue types have strong multi-scale directional properties, that are well captured by imaging protocols with high resolutions and spherical spatial transfer functions. The information modeled by the various image processing techniques is analyzed and visualized by displaying their 3-D texture primitives. We demonstrate that non-convolutional approaches are expected to provide best results when the size of structures are inferior to five voxels. For larger structures, it is shown that only multi-scale directional convolutional approaches that are non-separable allow for an unbiased modeling of 3-D biomedical textures. With the increase of high-resolution isotropic imaging protocols in clinical routine and research, these models are expected to best leverage the wealth of 3-D biomedical texture analysis in the future. Future research directions and opportunities are proposed to efficiently model personalized image-based phenotypes of normal biomedical tissue and its alterations. The integration of the clinical and genomic context is expected to better explain the intra class variation of healthy biomedical textures. Using texture synthesis, this provides the exciting opportunity to simulate and visualize texture atlases of normal ageing process and disease progression for enhanced treatment planning and clinical care management.

Rotation-Covariant Texture Learning Using Steerable Riesz Wavelets.

We propose a texture learning approach that exploits local organizations of scales and directions. First, linear combinations of Riesz wavelets are learned using kernel support vector machines. The resulting texture signatures are modeling optimal class-wise discriminatory properties. The visualization of the obtained signatures allows verifying the visual relevance of the learned concepts. Second, the local orientations of the signatures are optimized to maximize their responses, which is carried out analytically and can still be expressed as a linear combination of the initial steerable Riesz templates. The global process is iteratively repeated to obtain final rotation-covariant texture signatures. Rapid convergence of class-wise signatures is observed, which demonstrates that the instances are projected into a feature space that leverages the local organizations of scales and directions. Experimental evaluation reveals average classification accuracies in the range of 97% to 98% for the Outex_TC_00010, the Outex_TC_00012, and the Contrib_TC_00000 suites for even orders of the Riesz transform, and suggests high robustness to changes in images orientation and illumination. The proposed framework requires no arbitrary choices of scales and directions and is expected to perform well in a large range of computer vision applications.

Enhanced visualization of pulmonary perfusion in 4D dual energy CT images.

Pulmonary embolism (PE) affects up to 600,000 patients and contributes to at least 100,000 deaths every year in the United States alone. Diagnosis of PE can be difficult as most symptoms are unspecific. Computed Tomography (CT) angiography is the reference for diagnosing PE. CT angiography produces grayscale images with darker areas representing any mass filling defects, making the analysis of the images difficult. This article demonstrates a method using the combination of energy levels in Dual Energy CT images to highlight the presence of PE in the lung. The results show that pairing different energy levels from 40 to 140 keV can increase the contrast between well perfused areas and underperfused areas of the lung. In addition, the visualization used in the current study complies with the window/level settings usually employed by radiologists.

Benefits of texture analysis of dual energy CT for Computer-Aided pulmonary embolism detection.

Pulmonary embolism is an avoidable cause of death if treated immediately but delays in diagnosis and treatment lead to an increased risk. Computer-assisted image analysis of both unenhanced and contrast-enhanced computed tomography (CT) have proven useful for diagnosis of pulmonary embolism. Dual energy CT provides additional information over the standard single energy scan by generating four-dimensional (4D) data, in our case with 11 energy levels in 3D. In this paper a 4D texture analysis method capable of detecting pulmonary embolism in dual energy CT is presented. The method uses wavelet-based visual words together with an automatic geodesic-based region of interest detection algorithm to characterize the texture properties of each lung lobe. Results show an increase in performance with respect to the single energy CT analysis, as well as an accuracy gain compared to preliminary work on a small dataset.

Epileptogenic lesion quantification in MRI using contralateral 3D texture comparisons.

Epilepsy is a disorder of the brain that can lead to acute crisis and temporary loss of brain functions. Surgery is used to remove focal lesions that remain resistant to treatment. An accurate localization of epileptogenic lesions has a strong influence on the outcome of epilepsy surgery. Magnetic resonance imaging (MRI) is clinically used for lesion detection and treatment planning, mainly through simple visual analysis. However, visual inspection in MRI can be highly subjective and subtle 3D structural abnormalities are not always entirely removed during surgery. In this paper, we introduce a lesion abnormality score based on computerized comparison of the 3D texture properties between brain hemispheres in T1 MRI. Overlapping cubic texture blocks extracted from user-defined 3D regions of interest (ROI) are expressed in terms of energies of 3D steerable Riesz wavelets. The abnormality score is defined as the Hausdorff distance between the ROI and its corresponding contralateral region in the brain, both expressed as ensembles of blocks in the feature space. A classification based on the proposed score allowed an accuracy of 85% with 10 control subjects and 8 patients with epileptogenic lesions. The approach therefore constitutes a valuable tool for the objective pre-surgical evaluation of patients undergoing epilepsy surgery.

Multiscale lung texture signature learning using the Riesz transform.

Texture-based computerized analysis of high-resolution computed tomography images from patients with interstitial lung diseases is introduced to assist radiologists in image interpretation. The cornerstone of our approach is to learn lung texture signatures using a linear combination of N-th order Riesz templates at multiple scales. The weights of the linear combination are derived from one-versus-all support vector machines. Steerability and multiscale properties of Riesz wavelets allow for scale and rotation covariance of the texture descriptors with infinitesimal precision. Orientations are normalized among texture instances by locally aligning the Riesz templates, which is carried out analytically. The proposed approach is compared with state-of-the-art texture attributes and shows significant improvement in classification performance with an average area under receiver operating characteristic curves of 0.94 for five lung tissue classes. The derived lung texture signatures illustrate optimal class wise discriminative properties.

Lung texture classification using locally-oriented Riesz components.

We develop a texture analysis framework to assist radiologists in interpreting high-resolution computed tomography (HRCT) images of the lungs of patients affected with interstitial lung diseases (ILD). Novel texture descriptors based on the Riesz transform are proposed to analyze lung texture without any assumption on prevailing scales and orientations. A global classification accuracy of 78.3% among five lung tissue types is achieved using locally-oriented Riesz components. Comparative performance analysis with features derived from optimized grey-level co-occurrence matrices showed an absolute gain of 6.1% in classification accuracy. The adaptability of the Riesz features is demonstrated by reconstructing templates according to the first principal components of the lung textures. The balanced performance achieved among the various lung textures suggest that the proposed methods can complement human observers in HRCT interpretation, and opens interesting perspectives for future research.